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ABSTRACT 

Evaluating  the  capability  of  autonomous  infrared  (IR)  trackers  requires  tools  and  methods  to 
assess  tracker  performance  as  a function  of  target  and  background  conditions.  A methodology 
for  assessing  the  performance  of  such  IR  trackers  has  been  developed  and  implemented  for 
analyzing  captive  flight  test  IR  seeker  imagery.  Imagery  is  post-processed  using  the  Tracker 
Analysis  and  Ground-Truth  (TAG)  tool  to  establish  the  target  position  in  every  frame  of  an  IR 
sequence.  The  signature  metrics  of  the  target  and  background  are  extracted  from  each  frame 
using  the  Software  for  Extracting  Metrics  from  IR  Sequences  (SEMIRS).  The  sequences  are  then 
processed  using  the  Imaging  Seeker  Algorithm  Testbed  (ISAT)  to  assess  the  performance  of 
individual  trackers.  Finally,  software  has  been  implemented  to  match  tracker  performance 
results  with  the  extracted  signature  information,  producing  a spreadsheet  compatible  file  that  can 
be  used  for  analysis  and  plot  generating.  This  process  was  developed  and  demonstrated  using 
more  than  50  IR  sequences  of  a CFT  IR  seeker  in  closing  encounters  against  ground  vehicles. 


1.  Introduction 


The  evaluation  of  autonomous  IR  trackers  has  been  ongoing  for  many  years.  Recent  work  at  the  US  Army 
Aviation  and  Missile  Command  Research  Engineering  and  Development  Center  (AMRDEC)  has  involved 
analyzing  the  performance  of  multiple  IR  trackers  using  Captive  Flight  Test  (CFT)  data.  The  desire  has  been  to 
determine  the  correlation  between  tracker  performance  and  target  and  background  signature  metrics.  Dynetics  and 
AMRDEC  have  derived  a method  for  this  type  of  analysis  that  is  systematic  and  repeatable  for  any  number  of  IR 
trackers  or  targets  that  need  to  be  evaluated.  Though  developed  for  the  application  of  evaluating  CFT  data,  the 
methodology  is  applicable  to  evaluation  of  IR  trackers  using  synthetic  data. 

Until  recently,  extracting  the  target  and  background  signature  metrics  was  a tedious,  manually  intensive 
process  requiring  many  days  to  process  a single  sensor-to-target  engagement.  To  overcome  this,  Dynetics 
developed  the  TAG  tool  to  aid  in  ground-truthing  IR  image  sequences  (Reference  1).  Ground-truthing  is  the 
process  of  identifying  the  target  pixels  in  every  frame  of  an  IR  sequence.  The  TAG  tool  allows  ground-truthing 
entire  sequences  containing  4000  images  in  a matter  of  hours.  The  remainder  of  this  paper  will  present  the 
methodology  and  give  some  examples  of  its  implementation. 
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2.  Methodology  for  Evaluating  IR  Tracker  Performance 

The  approach  to  evaluating  IR  tracker  performance  is  shown  in  Figure  1.  The  process  starts  with  the 
actual  sequence  of  IR  images  taken  from  a CFT  closing  engagement  of  a seeker  with  a target.  Also  needed  are  the 
blackbody  imagery  and  physical  temperatures  that  are  used  for  calibrating  the  imagery.  Each  of  the  individual 
steps  of  the  methodology  is  addressed  in  the  following  subsections. 


Blackbody  IR  Imagery 

Sequence  & Temperatures 


Figure  1 . Methodology  for  Evaluating  Autonomous  IR  Tracker  Performance 
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2.1 


Generating  a Ground-Truth  File 


The  process  of  generating  a ground-truth  file  is  an  important  step  in  evaluating  an  IR  tracker.  The 
ground-truth  serves  three  purposes.  The  first  is  to  target  and  bacicground  metrics  in  an  automated  fashion.  The 
target  pixels  are  identified  within  a rectangular  box  that  a user  defines  using  the  TAG  Tool.  The  second  is  to 
generate  image  stabilization  information  that  can  be  used  by  the  tracker  in  the  event  that  there  are  large  jerks  from 
the  gimbals  in  the  CFT  data.  (The  CFT  data  is  typically  collected  open  loop,  and  depending  on  the  seeker  gimbals, 
pointing  joy  stick,  and  seeker  operator  interactions,  there  can  be  inadvertent  jerks  in  the  seeker  pointing.)  The 
third  purpose  of  the  ground-truth  is  to  enable  automatic  scoring  of  the  IR  tracker  performance.  This  is 
accomplished  by  comparing  the  location  of  the  IR  tracker  box  to  the  ground-truth  box. 

Generation  of  the  ground-truth  for  a single  IR  sequence  comprising  4000  frames  of  data  requires 
approximately  2-4  hours  of  time  are  required.  The  accuracy  of  the  ground-truth  box  location  and  size  is  very 
important  to  the  overall  process  including  both  the  signature  metrics  extraction  and  IR  tracker  scoring.  If  the  box 
is  too  big  or  is  not  centered  on  the  target,  then  the  target  signature  will  be  computed  on  a region  that  actually 
contains  target  and  background.  This  same  inaccuracy  of  the  target  box  size  or  its  placement  will  cause 
inaccuracies  in  the  automatic  scoring  of  the  IR  tracker.  Figure  2 shows  a ground-truth  box  that  is  appropriately 
sized  and  placed  on  the  target. 

To  avoid  these  inaccuracies,  great  care  is  taken  in  generating  the  ground-truth  using  the  TAG  Tool. 
Often,  the  user  will  start  ground-truthing  the  target  location  at  the  end  of  a sequence  at  which  point  the  seeker  was 
at  its  closest  approach  to  the  target.  The  user  then  typically  steps  backwards  through  the  sequence,  identifying  the 
target  in  each  image.  As  the  target  slant  range  increases  and  the  target  becomes  smaller,  the  user  is  better  able  to 
precisely  place  and  adjust  the  box  over  the  target  when  performing  this  operation  while  stepping  the  IR  images 
backwards.  Reference  1 gives  a complete  overview  of  the  TAG  Tool  and  the  steps  for  generating  the  ground-truth 
file. 


Figure  2.  Example  of  Ground-Truth  Box  Placed  on  Target 
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Computing  Calibration  Coefficients 

In  order  to  compute  signature  metrics  of  the  target  and  background,  calibration  information  must  first  be 
determined.  The  typical  process  is  to  relate  the  IR  image  gray  levels  to  temperature  with  a linear  transformation. 
Over  a small  enough  temperature  range  (20-30  K span),  the  linear  fit  is  satisfactory  for  this  transformation.  The 
Calibrated  Signature  Analysis  Tool  (CalSAT)  is  used  to  generate  a slope  and  intercept  that  transforms  the 
individual  pixels  of  an  IR  image  into  a temperature.  Typically,  the  user  imports  an  IR  image  with  the  four 
blackbodies  for  which  the  temperatures  are  known,  as  shown  in  Figure  3.  The  assumption  is  that  the  seeker 
viewed  the  blackbodies  in  the  field  during  the  CFT  and  close  in  time  to  the  actual  IR  sequence  to  be  analyzed. 
Also  important  is  that  the  seeker  digital  imagery  is  stored  without  any  automatic  gain  control  adjustments  to  the 
data.  Finally,  a non -uniformity  correction  for  a focal  plane  array  seeker  is  necessary.  CalSAT  will  compute  the 
average  gray  level  of  each  blackbody,  and  using  the  known  temperatures  of  each,  will  compute  a slope  and 
intercept  to  transform  the  gray  levels  into  temperature.  The  slope  and  intercept  are  stored  in  a file  for  later  use. 


Figure  3.  Test  Range  Scenario  for  Collecting  Calibration  Data 
2.3  Generating  Signature  Metrics  Using  SEMIRS 

The  next  step  after  generating  a ground-truth  file  and  calibration  coefficients  is  to  generate  signature 
metrics  for  every  image  of  the  IR  sequence.  The  Script  for  Extracting  Metrics  from  IR  Sequences  (SEMIRS)  was 
developed  to  accomplish  this  task.  The  inputs  to  SEMIRS  are  the  IR  sequence,  its  ground-truth  file,  and  the 
calibration  coefficients.  SEMIRS  then  generates  an  output  file  containing  many  standard  target  and  background 
signature  metrics  as  shown  in  Table  1.  In  every  case,  the  target  region  is  defined  as  a rectangle  (as  discussed  in  the 
previous  section)  and  the  background  region  is  an  area  that  is  twice  as  high  and  wide  as  the  target  area,  centered 
on  the  target  but  excluding  the  target  box.  The  metrics  shown  are  apparent  quantities,  not  source  measurements, 
because  this  is  typically  what  is  of  interest  in  assessing  the  IR  tracker  performance.  SEMIRS  is  executed  as  a 
command  line  entry  with  associated  arguments  making  it  well  suited  for  batch  processing.  Reference  2 gives  a 
more  in  depth  overview  of  the  metrics  shown  in  Table  1.  Reference  3 gives  more  details  on  calibrating  IR 
sequences  and  extracting  signature  metrics. 
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Table  1.  Signature  Metrics  Computed  by  SEMIRS 


Metric 

Definition 

T 

Atgt 

Average  Target  Temperature 

atci 

Standard  Deviation  of  Target  Temperature 

'Nlohe 

Number  of  Pixels  in  Target  Region 

Mtct 

Median  of  Target  Temperature 

Mjkp 

Average  Background  Temperature 

tfbkc 

Standard  Deviation  of  Background  Temperature 

l^BDix 

Number  of  Pixels  in  Background  Region 

Mbk-2 

Median  of  Background  Temperature 

AT 

> 

H 

II 

jr\  1 

'B 

! 

jf" 

ATrss 

ATrss  --\/aT2  +<?tgt 

SCRrss 

^Aj2+Gm 

SCRrss  - 

°bkg 

GLCM-TM 

Gray  Level  Co-occurrence  Matrix  - Trackability  Metrics 

2.4  Running  ISAT  RunRanges  to  Generate  Inputs  and  Execute  ISAT 

The  next  step  in  evaluating  the  IR  tracker  is  to  execute  a Perl  script  called  ISAT_RunRanges.  This  script 
prepares  the  input  files  needed  to  run  the  IR  tracker  using  the  Imaging  Seeker  Algorithm  Testbed  (ISAT).  ISAT  is 
a software  tool  for  design,  development,  and  evaluation  of  acquisition  and  tracking  algorithms  developed 
previously  by  Dynetics  for  AMRDEC.  The  specific  IR  tracker  under  evaluation  is  added  to  and  executed  within  the 
ISAT  framework.  An  initial  tracker  box  location  is  defined  by  the  user  after  which  the  tracker  autonomously 
determines  where  the  target  is  on  the  next  IR  frame  until  the  sequence  is  exhausted.  For  this  specific  application, 
the  initial  tracker  box  was  taken  from  the  ground-truth  file.  One  newly  added  feature  of  ISAT  is  that  it  can 
automatically  score  the  performance  of  the  tracker  by  comparing  the  tracker  box  location  to  the  ground-truth  box. 
This  is  discussed  in  the  next  subsection.  When  a track  is  terminated  either  because  the  sequence  was  at  its  end  or 
because  the  tracker  lost  track  of  the  target,  then  ISAT  writes  out  summary  information.  The  ISAT^RunRanges 
script  then  uses  this  summary  information  to  look  up  the  corresponding  signature  metrics  for  the  beginning  and 
end  of  the  run.  It  will  also  set  up  a new  run  at  a closer  slant  range  if  the  tracker  failed  to  maintain  track  on  the 
target  throughout  the  entire  run. 

2.5  Automatic  Scoring  of  the  Tracker  Performance 


A metric  was  developed  for  assessing  track  quality  or  track  error,  and  it  is  called  “Instantaneous  Track 
Error  Metric  - Overlap  Error”.  It  is  based  on  the  amount  of  overlap  between  the  ground-truth  and  tracker  boxes. 
Figure  4 shows  a tracker  box  overlaid  on  a ground-truth  box  in  an  IR  image  with  each  region  identified.  The  area 
(in  pixels)  for  the  truth  region,  tracker  region,  and  overlap  region  are  defined  as  NTRU,  NTRK,  and  NOL, 
respectively.  A ratio  of  NOL  to  NTRU  gives  an  indication  of  the  overlap  compared  to  the  ground-truth  box.  For 
the  example  in  Figure  4,  let  us  assume  that  this  ratio  is  approximately  0.45,  or  there  is  45%  overlap  between  the 
tracker  and  truth  boxes,  as  compared  to  the  truth  box.  Another  ratio  between  NOL  to  NTRK  is  defined  to  give 
some  indication  of  how  large  the  overlap  region  is  compared  to  the  track  box,  thus  giving  a measure  of  the  size  of 
the  track  box.  For  the  example  in  Figure  4,  let  us  assume  that  this  ratio  is  0.25,  or  there  is  25%  overlap  between 
the  tracker  and  truth  boxes,  as  compared  to  the  tracker  box.  If  the  track  box  is  extremely  large  compared  to  the 
truth  box,  this  second  ratio  will  be  much  smaller  than  the  first. 
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Figure  4 r.  Track  Box  Overlaid  on  Ground-Truth  Box  in  IR  Image 


Let  us  define  each  ratio  as: 


r>  _ 1N0L 
^TRU  “ 

^TRU 


R Nol 

TRK  “ N 

-L  ^ TDV 


By  averaging  the  two  ratios  we  can  derive  an  instantaneous  track  quality  metric  as: 

M_  -^tru  +^TRK 
Tfl 


This  metric  has  the  desired  characteristics  that  if  the  tracker  box  is  exactly  overlaid  on  and  is  the  same  size  as  the 
truth  box,  MTq  = 1.  If  there  is  no  overlap,  the  MTq  = 0.  To  convert  this  metric  into  a “error”  type  metric  where  a 
“1”  indicates  maximum  error  and  a “0”  indicates  no  error,  we  can  define  an  Overlap  Error  Metric  as: 

Moe  (4) 

As  ISAT  executes  the  tracker  algorithm,  it  scores  each  individual  IR  frame  using  this  overlap  error  metric. 
A criterion  of  Moe  of  0.95  or  greater  was  chosen  to  indicate  that  the  tracker  was  no  longer  tracking  the  target.  If 
more  that  10%  of  the  total  number  of  frames  of  the  sequence  exceeded  this  M0e  criterion,  then  the  target  was 
scored  as  “not  tracked”  by  the  tracker  algorithm. 

3.  Example  of  Methodology 


This  section  gives  an  example  of  using  the  methodology  to  evaluate  an  IR  tracker  performance  against  a 
T-72  tank.  Imagery  was  collected  from  a captive  flight  test  platform  using  an  imaging  IR  seeker.  The  T-72  tank 
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was  moving  throughout  the  run  and  was  being  viewed  by  the  seeker  from  a 270  decree  target  aspect  The  tracker 
evaluated  was  a simple  hot  spot  tracker.  The  initial  track  box  is  taken  from  the  ground-truth  data  for  this 
sequence,  and  the  track  was  initiated  at  approximately  3.000  m slant  range.  Figure  5 shows  representative  images 
from  the  sequence  at  closing  slant  ranges  with  the  ground-truth  box  overlaid  on  the  target.  The  ground-truth  box 
was  generated  using  the  TAG  tool.  Calibration  was  performed  for  this  sequence  using  CalSAT  alone  with  a 
blackbody  image  and  the  known  blackbody  temperatures. 


Figure  5.  Representative  JR  Images  of  Closing  Sequence  at  Various  Slant  Ranges 

SEMIRS  was  then  executed  to  extract  the  metrics  for  all  of  the  IR  frames  of  this  sequence.  Figure  6 
shows  a plot  of  ATRss,  <T.kg,  and  SCRRSs.  Note  how  the  apparent  ATRSs  increases  as  slant  range  decreases,  as 
expected,  because  there  is  less  atmospheric  attenuation  affecting  the  AT  and  there  are  more  pixels  on  target  which 
tends  to  increase  crtgl.  Figure  7 shows  a plot  of  the  target  signature  metrics  including  AT,  otgt , ATRSS,  and  the 
number  of  pixels  on  target.  The  first  three  parameters  are  plotted  using  the  left  Y-axis,  and  the  number  of  pixels  is 
plotted  using  the  logarithmic  Y-axis  on  the  right  side  of  the  plot. 

Figure  8 shows  a plot  of  the  overlap  error  metrics  (Moe)  as  a function  of  range  to  the  target.  This 
demonstrates  how  well  the  hot  spot  tracker  was  able  to  keep  the  track  box  centered  on  the  target.  Given  that  the 
hottest  portion  of  the  target  is  the  exhaust  towards  the  rear  of  the  tank,  the  tracker  box  was  never  completely 
centered  up  on  the  target,  therefore  there  is  always  some  overlap  error.  Figure  9 shows  the  tracker  and  ground- 
truth  boxes  for  one  IR  frame  of  this  sequence,  demonstrating  how  the  tracker  centered  its  box  on  the  hot  exhaust, 
as  expected. 
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Overlap  Error  Metric 


fie  View  Moctes  Qptsorn  Help 


Figure  9 . Tracker  Box  and  Ground-Truth  Box  Placements 
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The  ISAT_RunRange  perl  script  accumulated  the  summary  information  shown  in  Table  2 for  this 
particular  sequence.  The  temperature  metrics  are  in  units  of  degrees  Celsius.  As  a reminder,  all  values  were 
computed  as  apparent  quantities. 

Table  2 . Summary ' Signature  Metrics  at  Beginning  and  End  of  Run 


Metric 

Beginning  Value 

Ending  Value 

T 

Atgt 

20.1601 

30.8602 

<?tet 

1.5132 

5.7718 

84 

20436 

Abkg 

18.8259 

27.8563 

tfbk2 

0.4963 

2.8030 

I^Boix 

252 

55486 

AT  ’ 

1.3342 

3.0038 

ATrss 

2.0176 

6.5067 

SCRrss 

4.0881 

2.3213 

GLCM-TM 

0.8605 

0.8375 

By  repeating  this  process  for  multiple  seeker  passes  against  an  individual  target,  statistics  of  successful 
versus  unsuccessful  track  attempts  were  accumulated.  From  this,  analyses  of  signature  metrics  for  successful  tracks 
were  compared  to  signature  metrics  for  unsuccessful  tracks. 

4.  Conclusions 

The  methodology  presented  in  this  paper  was  used  to  process  over  50  different  1R  sequences  collected 
during  a CFT.  The  resulting  data  supported  analysis  of  IR  tracker  performance  as  a function  of  target  and 
background  signature  levels.  Performance  as  a function  of  initial  acquisition  range  was  also  derived  from  the  data. 
The  software  tools  were  developed  and  enhanced  to  support  the  processing  of  a large  volume  of  data  in  a relatively 
short  period  of  time.  The  most  labor-intensive  portion  of  the  processing  is  the  ground-truth  generation. 
Automation  of  the  tracker  processing  and  scoring  was  the  key  to  completing  the  analysis  on  this  large  data  set. 
The  software  tools  developed  by  Dynetics  for  this  particular  type  of  analysis,  including  TAG  tool,  SEM1RS, 
CalSAT,  ISAT,  and  ISATJRunRanges,  were  crucial  for  automating  as  much  of  the  process  as  possible. 
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